CLASSICAL AND SIGNED KAZHDAN-LUSZTIG POLYNOMIALS: 
CHARACTER MULTIPLICITY INVERSION BY INDUCTION 



WAI LING YEE 



Abstract. The famous Kazhdan-Lusztig Conjecture of the 1970s states that the multiplic- 
ity of an irreducible composition factor of a Verma module can be computed by evaluating 
Kazhdan-Lusztig polynomials at 1. Thus the character of a Verma module is a linear 
combination of characters of irreducible highest weight modules where the coefficients in 
the linear combination are Kazhdan-Lusztig polynomials evaluated at 1. Kazhdan-Lusztig 
showed that inverting and writing the character of an irreducible highest weight module as a 
linear combination of characters of Verma modules, the coefficients in the linear combination 
are also Kazhdan-Lusztig polynomials evaluated at 1, up to a sign. In this paper, we show 
how to prove Kazhdan-Lusztig's character multiplicity inversion formula by induction using 
coherent continuation functors. Unitary representations may be identified by determining 
if characters and signature characters are the same. The signature character of a Verma 
module may be written as a linear combination of signature characters of irreducible highest 
weight modules where the coefficients in the linear combination are signed Kazhdan-Lusztig 
polynomials evaluated at 1. An analogous argument by induction using coherent continu- 
ation functors proves an analogous multiplicity inversion formula for signature characters: 
the signature character of an irreducible highest weight module is a linear combination of 
signature characters of Verma modules where the coefficients, up to a sign, are also signed 
Kazhdan-Lusztig polynomials evaluated at 1. 



1. Introduction 

Finding the composition factor multiplicities for a Verma module was one of the premier 
open problems in representation theory in the 1970s and the early 1980s. Kazhdan and 
Lusztig conjectured in their seminal paper |KL79] that the multiplicities equaled Kazhdan- 
Lusztig polynomials evaluated at 1. Jantzen's Conjecture provided even deeper insight into 
the structure of a Verma module: the various coefficients of the polynomials gave multiplic- 
ities of an irreducible composition factor within particular levels of the Jantzen filtration of 
the Verma module. Brylinski-Kashiwara and Beilinson-Bernstein independently proved the 
Kazhdan-Lusztig Conjecture. Subsequently, Beilinson-Bernstein proved Jantzen's Conjec- 
ture in [BB93j . 

The most common approach to classifying unitary representations is to first identify rep- 
resentations which admit invariant Hermitian forms, compute the signatures of those forms, 
and then determine which forms are definite. The signature of the Hermitian form may 
be stored as a signature character, and then determining unitarity is equivalent to testing 
if the character and the signature character are the same. See |Wal84] . [Yee05j . [Yee08j. 
and |Yeej for work on signature characters for irreducible highest weight modules. Because 



1991 Mathematics Subject Classification. Primary 22E50, Secondary 05E10. 

The author is grateful for the support from a Discovery Grant and UFA from NSERC, and NSF grants 
DMS-0554278 and DMS-0968275. 

1 



understanding the structure of each level of the Jantzen filtration is crucial to computing 
signatures of invariant Hermitian forms, signed Kazhdan-Lusztig polynomials were defined 
in |Yee08] . which store signature information for each level of the Jantzen filtration in its 
coefficients rather than multiplicities. 

Consider g, a complex semisimple Lie algebra, and f), a Cartan subalgebra of g. Choose 
positive roots, let p be the half sum of the positive roots, and let A G f)* be antidominant. 
Let g = n © h, © n~ be the triangular decomposition and b = t) © n the Borel subalgebra 
determined by the choice of positive roots and let M(p) = U (g) ®u{b) C M _ P . Let W\ be the 
integral Weyl group and w° x its long element. Given x G W\, composition factors of M(x\) 
are of the form L(yX) where y G W\ and y < x. The multiplicity of L(yX) in M(xX) is given 
by 

[M(xX) : L{yX)\ = P w o X)<y (l) 

from which it follows that 

chM(xX) = ^2P w o xXW o xy (l)chL(yX). 

y<x 

Kazhdan and Lusztig's inversion formula indicates that 

chL(xX) = Y,{- l Y {x) ~ m PvA l )chM{yX). 

y<x 

In |Yee08] . signed Kazhdan-Lusztig polynomials were defined in order to compute signa- 
tures of invariant Hermitian forms. Assume g and f} to be complexifications of go D f)o 
and assume E)o to be stable under 9, the Cartan involution of go- Then M(p) admits an 
invariant Hermitian form if and only if 9(b) = b and p is imaginary. Invariant Hermitian 
forms on M(p) are unique up to a real scalar. The canonical form is called the Shapovalov 
form. Its radical is the unique maximal proper submodule of M(p), and hence it descends 
to a canonical invariant Hermitian form on L(p). Assume the real form to be equal rank. 
Then 9(b) = b implies that all of the roots are imaginary. Assume that A is imaginary and 
antidominant. Then the signed Kazhdan-Lusztig polynomials satisfy for small t > 0: 

e w ^ t ch s M(xX + w(-p)t) = J2P^ x , <y (l)ch s L(yX). 

y<x 

The inversion formula providing an expression for ch s L(xX) as a linear combination of 
ch s M(yX + w(—p)t) found in |Yee 08j is cumbersome. Particularly in light of a close re- 
lationship proved in |Yeej between classical and signed Kazhdan-Lusztig polynomials, we are 
able to prove by induction using coherent continuation functors in the equal rank case the 
following formula, which is analogous to the inversion formula for characters: 
Main Theorem: 

ch s L(xX) = e< {p)t ^(-lY {x) - £{y) Py;f\l)ch s M(yX + wl(-p)t). 

y<x 

Here, e is the Z 2 -grading on the root lattice with e(p) being equal to the parity of the number 
of non-compact roots in an expression for p as a sum of roots. 
The paper is structured as follows. 

The first half of the paper deals with classical Kazhdan-Lusztig polynomials. First, we 
review the relationship between Kazhdan-Lusztig polynomials and the Jantzen filtration 
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in section 2. We then review the coherent continuation functor in section 3. In section 

4, we prove Kazhdan and Lusztig's character multiplicity inversion formula by induction 
using coherent continuation functors. Just as Gabber and Joseph showed in |GJ81] that 
Jantzen's Conjecture and coherent continuation functors gave recursive formulas for comput- 
ing Kazhdan-Lusztig polynomials, permitting the stepwise computation of Kazhdan-Lusztig 
polynomials, applying coherent continuation functors allows us to prove a character multi- 
plicity inverison formula by induction. 

The second half of the paper concerns signed Kazhdan-Lusztig polynomials. In section 

5, we review signature character theory for highest weight modules, the Jantzen filtration, 
and the definition of signed Kazhdan-Lusztig polynomials. In section 6, we discuss coherent 
continuation functors in the enhanced setting used to compute signature characters. In 
section 7, we apply the same induction argument to signature characters in the case where 
the real form is compact to obtain an inversion formula. In section 8, we show how a simple 
change of basis argument provides an inversion formula for other equal rank real forms. 

Section 9 contains a description of future work. 

1.1. Acknowledgements: David Vogan and Gregg Zuckerman discussed using the coherent 
continuation representation of the Weyl group to compute characters of irreducible Harish- 
Chandra modules at the 2010 Atlas of Lie Groups workshop. Building representations using 
coherent continuation is the philosophy used in this paper. 

2. Kazhdan-Lusztig Polynomials and Jantzen's Conjecture 
We use the following setup in the first half of this paper: 

Notation 2.1. - g is a complex semisimple Lie algebra 

- t) is a Cartan subalgebra 

- A(g, ()) is the set of roots 

- A + (g, f)) is a choice of positive roots 

- p is one half the sum of the positive roots 

- LT is the set of simple roots 

-g = n©t)©n~is the corresponding triangular decomposition 

- b = t) © n is the corresponding Borel subalgebra 

- A r is the root lattice 

- A is the integral weight lattice 

- A G f)* is antidominant 

- W\ is the integral Weyl group and u>° its long element 

- A A := {a G A(g, f)) : (A,ct v ) G Z}. It is a root system with Weyl group W\. 

- LI A is a set of simple roots for A A determined by p 

- Given p G ()*, M(p) := U (g) <S>u{b) ^^-p is the Verma module of highest weight p — p. 

- v^p the canonical generator of M(p). 

- L(p) is the quotient of M(p) by its unique maximal proper submodule. It is an 
irreducible highest weight module. 

We begin by reviewing the Shapovalov form and the Jantzen filtration. Let o be the 
unique involutive automorphism of g such that 



a(X a ) = Y a , a(Y a ) = X a , a(H a ) = H ( 
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where a G A + (g, f)) and X a G Q a , Y a G $J_ Q , and G I) are Chevalley generators of g. Then 
there exists a bilinear form (•, •) : M(/i) x M(/i) — >■ C that is contravariant: contravariant 
means that 

(x • u, v) — (u, a(x)v) Wx G g and u, v G M(//). 
Normalized so that (t> Ai _ p , t> = 1, the form is unique and called the (classical) Shapovalov 
form. We denote it by (■, •) Ai . This form exists for every Verma module. 

Definition 2.2. Let V be a finite-dimensional vector space and let (•,•)* be an analytic 
family of bilinear forms on V where t G (—5, 5) and the forms are non-degenerate for t ^ 0. 
The Jantzen filtration is defined to be: 

V = V® D V {1) D • • • D V {n) = {0} 

where v G V" 1 ™-* if there is exists an analytic map j v : (— e, e) — >■ V for some e > such that: 

(1) 7«(0) = f and 

(2) for every u 6 K, (7u(£)>' u )t vanishes at least to order n at t = 0. 

Since (/i • it, u)^ = (u,h • v)^ for all /i G f) and u,v & M(/i), therefore the weight space 
decomposition is an orthogonal decomposition into finite-dimensional subspaces. Under the 
canonical identification of any M(/i) with U(n~), it makes sense to take an analytic path of 
highest weights equal to /i at t = and to consider the corresponding Jantzen filtration on 
M(/i). It was proved by Barbasch in |Bar83] that the Jantzen filtration does not depend on 
the analytic path chosen. 

Let A be antidominant and let x G W\. It is well-known that the j th level of the Jantzen 
filtration, M(x\) j := M(x\) {j) / M(x\)^ +1 \ is semisimple. It is a direct sum of irreducible 
highest weight modules of the form L(yX) where y G W\ and y < x. Jantzen's Conjecture 
gives the multiplicity of L(yX) in the j th level of the Jantzen filtration. 

Theorem 2.3. f |BB93j ) Jantzen's Conjecture: Let A be antidominant andx,y G W\. Then: 

[M{x\)i : L{y\)\ = coefficient of q W*)-'W->V* m P <x , <y (q). 

To compute Kazhdan-Lusztig polynomials, one may use the following formulas: 

Theorem 2.4. Let A be antidominant. Kazhdan-Lusztig polynomials may be defined by 
Px,x(q) — 1 f or a tt x £ W\ and P Xty (q) = for x, y G W\ and x % y along with the recursive 
formulas where s is a simple reflection: 

a) P x>y (q) = P xs ,y(q) ifys<y and x,xs <y 
a') P x , y {q) = Psx, y (q) ifsy<y andx,sx<y 

b) If y < ys, then: 

q c P xs , y {q) + q l ~ c P X)V (q) = ^ a *,vAQ ^ *' + * p xM + P x,ys{q) 

zs<z 

where c = 1 if xs > x and c = if xs < x. 

3. Coherent Continuation Functors 

We begin by recalling the definition of Category O and related concepts discussed in 
[BGG76] . 

Definition 3.1. Category O is the category of 0-modules V satisfying: 
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(1) V = @^*v, 

(2) V is finitely-generated over U (g) 

(3) V is n-finite: dim[/(n)i> < oo for every v G V. 

Verma modules and irreducible highest weight modules form two different bases of the 
Grothendieck group of Category O. 

If z G Z(g), the centre of the universal enveloping algebra, then the centre acts by scalars 
on Verma modules, so there is a map X/j. '■ Z(g) ~~ ^ ^ such that z • v — X/j.( z ) v f° r every 
z G Z(g) and every v G M(/i). Specifically, consider the decomposition U(g) = C/(f)) © 
(C/(fj)n + n~t/({j)) and let pr : U(g) ^ U{\)) be projection with respect to that direct sum. 
Then: 

Xr(z) = n(pr(z)) VzeZ(g). 
Xfj, is called a central character. x M = Xu if and only if v = wji for some element w of the 
Weyl group. 

Definition 3.2. Category O may be decomposed by central character into what are called 
blocks: 

£> M : = {V G O : 3 N such that (z - x^))^ annihilates V\/z G 
For example, M(xA) and L(x\) belong to 0\ for all x <E W. 
Definition 3.3. Decomposing category O by infinitesimal character: 

By Pr M , we mean projection onto (9 M under the above direct sum. Decomposition under the 
above direct sum is called primary decomposition. 

It is known that primary decomposition is an orthogonal decomposition. 
To define coherent continuation functors, we need Jantzen's translation functors which 
allow us to translate modules from one block of Category O to another. 

Definition 3.4. Choosing F(fi) the finite-dimensional representation of extremal integral 
weight fi, then 

V h. Pr x+ ^V®F(n)) 

is Jantzen's translation functor from 0\ —¥ O^. By projecting first onto 0\, the translation 
functor may be applied to all of Category O: 

V ^ Pr x+lt (Pr x (V)®F(n)). 

Now we recall the definition of coherent continuation functors: 

Definition 3.5. Let A G h* be antidominant regular and let s = s a G W\ be a simple 
reflection. There exists an integral weight u a such that A — u a is antidominant and the only 
simple root (5 for which (A — u a , j3) = is (5 = a. Then: 

(1) T x x ~ Ua is translation to the a-wall, 

(2) T^_ v is translation from the a-wall, and 
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(3) 9 a = T£_ u o Ua is the coherent continuation functor across the a-wall or the 
reflection functor across the a-wall. 

We recall the following facts about the coherent-continuation functor: 

Theorem 3.6. Let X be antidominant regular, a G 14^, and let s = s a . If x G W\, then: 

(1) 6 a M(x\) = 6 a M(xs\) 

(2) If x < xs, there is a short exact sequence 

-> M(xsX) -> 9 a M(xX) -» M(x\) 0. 

(3) If x > xs, then 6 a L(xa) = 0. 

(4) If x < xs, then 6 a L(xa) ^ and it has a unique simple quotient and a unique simple 
submodule, both isomorphic to L(x\). 

(5) If x < xs, then there is a chain complex 

->• L{xX) ^ 6 a L(xX) -» L{xX) ->■ 
whose cohomology is denoted by U a L(xX). Letting a w o x w o y j denote the coefficient of 

l(x)-l(y)-j 

q 2 in P w o XtW o xy (q), 

chU a L(xX) = ch L(xsX) + a w o^ x w o^ y l chL(yX). 

yeW x \y>ys 

These are the formulas we will use to prove the character multiplicity inversion formula 
by induction. 

4. Kazhdan-Lusztig Polynomials: Character Multiplicity Inversion 

The goal of this section is to prove Kazhdan-Lusztig' s character multiplicity inversion 
formula by induction: 

Theorem 4.1. ( |KL79j ) Let X G h* be antidominant and let x G W\. Then: 

chL(xX) = ^(-l)^P w (l)(AM(yA). 

y<x 

Proof. We prove this by induction on x. When x = 1, M(A) = L(X) while P 11 (g) = 1, so 
the formula holds. 

We suppose now by induction that for all z < x, 

chL(zX) = Y^(- l Y {z) ~ t{w) p w ,z(l)chM(wX). 

w<z 

In particular, for some s = s a simple, x > xs. By induction, 

chL(xsX) = Y,{-iy^- e W P y!XS (l)chM(yX). 

y<xs 

Following an idea of Gabber and Joseph, who showed how to build recursive formulas for 
computing Kazhdan-Lusztig polynomials using coherent continuation functors assuming that 
Jantzen's Conjecture holds, we apply 9 a to both sides of this equation and get: 

ch6 a L(xsX) = J2{-lf xsym P y , xs (l)che a M(yX). 

y<xs 
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The equations of Theorem 13.61 now imply: 

2chL(xs\)+chL(x\) + J2 a w 1xs, w <iz,ichL(z\) = P yjXS (l) (ch M(y\) + ch M(ys\)) . 

z~>zs yi£xs 

Applying the induction hypothesis to chL(xsX) and chL(zX) and rearranging, we see that 
chL(xX) = ^(-1)^ m )-^P w , m (1)(-c^M(j/A) + c^M(j/sA)) 

y<xs 

- £ £(-l)^ MW P^(l) M(t«A) 

Z>.ZS U><2 

To prove the theorem, we study the coefficient in front of a fixed chM(w\) where w < x. 
Let c(M(wX)) denote this coefficient. 

Since x > xs, by Property Z, x > w, ws. 

Case 1): Both w,ws < xs. 
The coefficient of M(wX) is: 

c(M(wX)) = (-l)^)-^)p^ s (l)(-l) + (-l)^)-«P u;S)a;s (l) 

— a w { lxs,w° x z,l(~ ^ W ^P W)Z (1) 

z>zs,z>w 

= (-i)^ ) -^ ) (p^(i)+p ro (i))- E m-^c- 1 )^^^ 1 )- 

Now a w o xs u ,o 2 l is the coefficient of g 2 in P w o XStW o z , and thus £(x) and £{z) have the 
same parity if a w o XSjW o z> i 7^ 0. We may replace (— l)^) - ^ ) with (— Furthermore, 
it is well-known that a w o xs>w o Zjl = a Z)XSj i (see, for example, |Hum90] . p. 165). Thus: 

c(M(wX)) = (-l) e(x) - e(w) (p w , xs (1) + P ws , xs (l) ~ £ o*,x-,i^,*(l)) 

= (_l)^)-^)P WtX (l) 

by formula b) of Theorem 12.41 

Case 2): x > w but xs ^ u>: 
Recall by Property Z that both w,ws < x. Also, we must have w > ws and xs > ws by 
Property Z. The coefficient of M(wX) is: 

C (m(™a)) = (-iy^-^p ws>xs (i)- j2 ^s^i-iy^-^p^) 

z>zs,z>w 

= (-1) i(x) - e{w) ( - E M-.^i^^ 1 )) 

= (P W (1)) 

since a^o^ ^ ! 7^ implies that 2 < xs while P WjZ 7^ implies that w < z. There are no z 
so that w < z < xs, so the sum is zero. 
It is known that the right side is now 

(_l)<(*M«)p ui (l) ( S ee, for example, [dC02] formula 

4.1 c)). 
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Case 3): xs > w but xs ^ ws: 
In this case, w < ws < x by Property Z. The coefficient of M(w\) is: 

c(M(wX)) = -(-l)Y {xs) '^P w ^)- E (" 1 ) l( * M(w) 'M«*i^( 1 ) 

z>zs,z>w 

( 1) ^ ^ ^ ^ I ~l" -P«is,x-s(l) ^ ^ ^z,a;s,l-P«),2(l) J since P X s,ws 

\ z>zs,z>w / 

In all other cases, the coefficient is 0. 
We see that in all cases, 

y<x 

proving Kazhdan-Lusztig's character multiplicity inversion formula by induction. □ 

5. Signed Kazhdan-Lusztig Polynomials and Jantzen's Conjecture 
We specialize the setup from the first half of the paper for the second half of this paper: 

Notation 5.1. - go is an equal rank real semisimple Lie algebra 

- 9 is a Cartan involution of go 

- 0o = £o © Po is the corresponding Cartan decomposition 

- ()o is a 6*-stable Cartan subalgebra of g 

- drop the subscript to denote complexification 

A Hermitian form (-, •) on a g- module V is invariant if 

(X ■ u,v) + (u, X ■ v) = 

for all X G g, where complex conjugation is with respect to the real form g of g. 

In our setup, the Verma module M(A) accepts a non-degenerate invariant Hermitian form 
if and only if 8(b) = b and A is imaginary ( [Yee05| . p. 641). Since g is equal rank, therefore 
P) is compact and all of the roots are imaginary. The form is unique up to a real scalar. If 
it is normalized so that (v\_ p ,v\^ p ) = 1, then it is called the Shapovalov (Hermitian) form 
and denoted by (•, -) A . 

Since the Chevalley basis vectors satisfy — X a = Y a if a is compact while — X a = —Y a if 
a is non-compact (see |Yee05] . 5.2.17), therefore (•, -) A and (•, -)\, the invariant Shapovalov 
Hermitian form and the contravariant Shapovalov bilinear form, are related. The weight 
space decomposition is an orthogonal decomposition into finite dimensional subspaces (see 
[Yee05j . p. 643) and just as in section [2] we can discuss the signature of the Shapovalov 
form. We can also discuss the Jantzen filtration on that Verma module by considering an 
analytic path of imaginary highest weights. The Jantzen filtration of a Verma module arising 
from invariant Hermitian forms and the Jantzen filtration of a Verma module arising from 
contravariant forms are the same. We denote the levels of the filtration with M(A)^ or 
with M(\)^ interchangeably. Furthermore, the Shapovalov form descends to an invariant 
Hermitian form on L(X). We can also discuss the signature for the highest weight module 
L(A). The form is non-degenerate, and signatures for each of the weight spaces can be stored 
in a signature character: 
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Definition 5.2. The signature character of a non-degenerate invariant Hermitian form (•, •) 
on a highest weight module V whose weights are A(V) is 

ch s (-,-) = ch s V= £ (j>(ri-q(n))eP 

MGA(V) 

where the signature on is (p(jtt), <?(a*))- 

The general approach to computing signature characters has been as follows. The Shapo- 
valov form on a Verma module is non-degenerate if and only if the Verma module is irre- 
ducible. M(A) is reducible if and only if (A, a v ) G Z + for some positive root a, so M(A) is 
irreducible away from various affine hyperplanes. Within any region bounded by these so 
called reducibility hyperplanes, the Shapovalov form cannot become degenerate, hence the 
signature is constant. The antidominant Weyl chamber does not contain any reducibility hy- 
perplanes. Within the reducibility hyperplane-free region containing the antidominant Weyl 
chamber, an asymptotic argument gives the signature character (see |Wal84] ) . In other re- 
gions, an induction argument and combinatorial analysis to determine how signatures change 
as you cross reducibility hyperplanes to travel from one region to another give the signa- 
ture character for other irreducible Verma modules (see [Yec08j). If M(A) is reducible, we 
wish instead to compute the signature of L(X). Such formulas were determined in |Yee08j 
and use so-called signed Kazhdan-Lusztig polynomials. We review the definition of signed 
Kazhdan-Lusztig polynomials. 

Let A be antidominant and imaginary and let x G W\. The j th level of the Jantzen 
filtration of M(xX) is semisimple and equal to the direct sum of modules of the form L(yX) 
where y G W\ and y < x. While Kazhdan-Lusztig polynomials record j th level multiplicities 
in their coefficients, signed Kazhdan-Lusztig polynomials were defined to record signatures. 
First recall that the Jantzen filtration naturally provides forms on each filtration level: 

Definition 5.3. Let A be antidominant, let x G W\, and let S G f)* be regular and imaginary. 
There is an invariant Hermitian form on M{xX)V'\ 

(u,v). := lim - ( 7tt (t), j v (t)) xX+st \/u,ve M(xX). 

This form descends to a non-degenerate form on M(xX)j = M(xX)^' /M(xX)^ +1 ' which we 
will also denote by (•, •) -. 

Remark 5.4. Note that the signatures of the forms (•, •) ■ depend on the direction 5 of the 
filtration. For any two directions belonging to the interior of the same Weyl chamber, there 
are no reducibility hyperplanes separating xX plus the direction vectors, so the signatures of 
(•, •) ■ are the same for those two choices of direction vector. Without loss of generality, we 
take our direction to be some w(—p) where w G W. 

Definition 5.5. Let A be antidominant, let x G W\, and w G W. Let the Jantzen filtration 

on M(xX) arise from the analytic family of invariant Hermitian forms (•, ') x \ +w (- p -\f Recall 

the non-degenerate invariant Hermitian form (•, •) . defined on M(xX)j. Let y G W\. Recall 

that M(xX)j is a direct sum of irreducible highest weight modules of the form L(yX). The 
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constants denned to satisfy: 

cK (-, -) 3 = ^a%, w o xV jCh s L{y\) 

y<x 

where by ch s L(y\) we mean the signature of the Shapovalov form. Signed Kazhdan-Lusztig 
polynomials are defined by: 

wXx,U)\y X1 ' / j w^x,w^y,j 1 
j>0 



By Proposition 3.3 of |Vog84 , the signature character for small t > is equal to the sum 



Ylj c hs (•> •),• °f a ^ the jth-level signature characters. Therefore: 
Proposition 5.6. For small t > 0: 

e*"* ch s M(xX + w(-p)t) = ch s (•, •) . = J2 P ^, <y (i)chsL(y\). 

j y<x 

The left hand side is known by work of |Yee05j while we would like to obtain a formula for 
ch s L(xX). In |Yee08] . this formula was inverted to give a formula for ch s L(xX) as a sum of 
signature characters of irreducible Verma modules. The formula was unwieldy and required 
two improvements: 

(1) The coefficients in the linear combination required simplification. We prove in this 
paper that the coefficients are, up to a sign, signed Kazhdan-Lusztig polynomials 
evaluated at 1. 

(2) Signed Kazhdan-Lusztig polynomials are equal to classical Kazhdan-Lusztig polyno- 
mials evaluated at — q rather than q and multiplied by a sign (the main theorem of 

ESI)- 

The following recursive formulas are satisfied by signed Kazdhan-Lusztig polynomials: 

Proposition 5.7. (Theorem 4-6.10 of [Yee08j . corrected in the proof of |Yee] Theorem 4-6) 
Let X G f)* be antidominant. Signed Kazhdan-Lusztig polynomials may be computed using 

Px'x x {q) = 1, Px^y x {q) = if x ^ y, and the following recursive formulas where s = s a 
where a G H\: 

a) If ys > y and xs > x > y then: 

P A ' W ° (n) = (_iy((X,<x w )xa)p\™l , \ = f 1 Y((xX,x a ^)xa)p^ ( i / n 

a') If sy > y and sx > x > y then: 

P \< „ (g) = (_l)«((«A,« V )«)p^ Q 
w^x.w^y^' v ' w^sx,w^y y ^' 

b) If y > ys and x < xs and x > y then: 

_(_l)^W )2a)p A r ° {q)+qP \< () (q) = q y a X >< o q «t^p*Z {q) 

v ' w^xs,w~^y y 1 ' 1 w^x,w^y yl/ a / j w^z^wix) 1 w^x,w^z yrl/ 

zEW\\z<zs 

_/ _-^y((ys\,ysa v )ysa) p^ w °\ / \ 
v ' w^XjW^ys^" ' 
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6. Gabber and Joseph's Enhanced Setting for Coherent Continuation 



Recall that Gabber and Joseph showed in [GJ81J how to use coherent continuation func- 
tors and Jantzen's Conjecture to prove recursive formulas for Kazhdan-Lusztig polynomials. 
In order to study how coherent continuation affects signature characters and to study signed 
Kazhdan-Lusztig polynomials, we need to use Gabber and Joseph's generalization of Cate- 
gory O. 

Our setup for this section is as follows: 

Notation 6.1. - A — C[i]( t ), a local ring 

- B := {X a , Y a : a G A + (g, f))} U {H a : a G 11} is the Chevalley basis for q 

- Qz = span z £ 

- f)z, bz, riz etc. are defined similarly 

- Qa ■■= Qz ®z A 

- f)^, ^a, tiA etc. are defined similarly 

- Given A G i)* A , Ax is the one- dimensional f}A-module where lie I] acts by X(h) 

- Given A G t)% M(X) A = U(g A ) ® bA A x _ p 

Gabber and Joseph make the following generalization of Category O: 

Definition 6.2. Let A G t)* A and let C C t)* A be of the form A + A, where A is the integral 
weight lattice. Then category Kc is the subcategory of U(qa) modules V such that: 

(2) V is U (n^-fmite (i.e. diuu U (xia) ■ v < oo for all v G V) 

(3) V is finitely generated over U (qa) 

The Verma modules M(A)a are objects in Kc- Each M(X)a has a unique maximal proper 
submodule. In general, for every maximal ideal m of A, there is a unique maximal proper 
submodule of M(A)a containing i7iM(X)a- The corresponding simple quotient is denoted 
by L(m,X). We have chosen A to be a local ring, so the maximal ideal of A is unique 
and the maximal proper submodule of M(X)a is unique, and we may write L(X) A for the 
corresponding simple quotient without ambiguity. These ^(A^'s are the simple objects of 
category Kc- 

Notation 6.3. Recall that we are working over the local ring A = C\t]r t y We denote by 7 
specialization at t = 0. That is, given any A-module V, V = V/tV. So, given A + 5t G t)* A , 
where A, 5 G ()*: 

- A + 5t = A, 

- M(A + 5t) A = M(A), and 

- L(X + 5t) A = L{X). 

Definition 6.4. If D C C, then category Ko is the subcategory of Kc whose simple 
subquotients are L(X)as where A G D. 

Definition 6.5. In the context of category Kc, a subset D C C is a block if there is some 
\i G f) A and 8 G f)* such that D = W^n + St. Given a block D, the following definitions may 
be made: 

(1) J D := n^okerxn 

(2) For V G ObK Cj Pr D V := {v G V : Vz G J , 3 N G Z+ such that z n v = 0}. 
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(3) V = Pr D .V is the primary decomposition of V. 

(4) Let Kd be the subcategory of Kc such that V is an object in Kd if and only if 
JpV = for large enough N. Pr^ takes objects in Kc to objects in Kd- 

Gabber and Joseph made the following extension of Jantzen's translation functors to 
category K c : 

Definition 6.6. Given A G f)* antidominant and regular and S G t)* regular, let D = 
W\X + St. Given p G A, let D' = W\(X +/i) + 8t and let F(p) A denote the finite-dimensional 
g^-niodule with extremal weight p. Then 

Tg'V = Pr D >(F(p) A ® A (Pr D V)) 

is Jantzen's translation functor from D to D' . 

This leads to their extension of the defintion of coherent continuation functors: 

Definition 6.7. Let A G f)* be antidominant regular, 5 G f}* be regular, D = W\\ + 5t, 
and let s a G W\ be a simple reflection. There exists an integral weight u a such that A — u a 
is antidominant and the only simple root (3 for which (A — v a , ff) = is (3 = a. Let 
D a = W X {\- v a ) + St. Then: 

(1) Tp™ is translation to the a- wall, 

(2) Tp a is translation from the a- wall, and 

(3) 9 a = Tg oTfl" is the coherent continuation functor across the a- wall or the reflection 
functor across the a-wall. 

Notation 6.8. Given A G 1)* regular antidominant, x G W\, and 5 G f)* regular, by 
ch s M(x\ + St) or ch s L(x\ + St) we mean the limiting signature on the specialized mod- 
ule as t — > + . 

For example, we may rewrite Proposition 15.61 as 

ch s M(xX + w(-p)t) = Y J PX^lv chsL ^- 

y<x 

We have the following results for signature characters and coherent continuation: 

Theorem 6.9. Let A G t)* be antidominant regular, a G rL\ ; and let s = s a . Let D = 
W\\ + w®(—p)t and let D a = W\(\ — u a ) + p)t be a block so that T^ a is a translation 
functor to the a-wall. Let A^ denote the highest weight of F(v a ) and X~ denote the highest 
weigt of F(—u a ). Then: 

(1) If z G W\ and z < zs, then: 

ch s e a M{z\ + w° x {-p)t) = _(_i)^((W)^) c/ls a M(zs\ + w° x {-p)t) 

= (_ 1 )e(A++A- +2 (A,aV )a ) c ^ + w 0(_ p ) t ) 

_(_1)<A++A-) C ^ M(z\ + W° x {-p)t) 

(2) If z > zs, then ch s 8 a L(z\) = 0. 
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(3) If z < zs, then 

ch s 9 a L(zX + w° x (-p)t) = {-l)< xi+x " +z ^ aV ^ch s L{zsX) 

y>ys 

Proof. To prove (JTJ, we begin by proving that if z < zs, 

ch s T° a M{zX + w° x {-p)t) = -{-l) e « x ' aV ^ch s TE«M{zs\ + w° x {-p)t) 

= (-l) e ^ +zv ^ch s M{z{\ - u a ) + w° x (-p)t) 
From the formulas at the bottom of p. 183 of [Ycc08j, for every z, 

ch s TE a M(zX + w° x {-p)t) = sgn(c' z )ch s M{z{\ - u a ) + w° x {-p)t). 
The formula for c' z from the top of p. 184 of |Yee08j is 

c' z = (-l)«*=+"«"> D F( _ Va) (-zu a ) a'_ tVa . 

Lemma 4.6.7 of [Yee08j states that the second term in the product always has sign 1 while 
Lemma 4.6.9. (i) of Yee08j states that the third term in the product has sign 1 if z < zs. 
Thus sgn(c 2 ) = (— l) e (*<* jf z < zs _ if z > zs, we must determine the sign of the third 
term, sgo.(af_ zv ). Follow the proof of 4.6.9. (i) exactly up to the last sentence. Then note that 
since (— zv a , za v ) < 0, therefore sgn(a'_ Z! , Q ) = —1. Thus sgn(c' 2 ) = — (— \y^ +zv ^i if z > zs; 
or if z < zs, sgn(c' 2 J = — (— i) e ( A a+ 2S ^). From this and from zv a — zsv a = z(X,a y )a)a, it 
follows that ch s 6 a M(zX + w° x (-p)t) = -(-l)<^' aV ^ch a 6 a M(zs\ + w° x (-p)t). 

From Theorem 13.61 we see that ch s 8 a M(z\ + w x (—p)t) must be a linear combination of 
ch s M(zX + w x (—p)t) and of ch s M(zsX + w x (—p)t). The discussion on the bottom of p. 184 
and the top of p. 185 of |Yee08j shows that for z < zs: 

ch s 6 a M(zX + w° x (t)) = (-1)^— -) sgn(D F(l/a) (zv a )al a c' z ) ch s M(z\ + w° x (-p)t) 

+ ( _ ir (A+-^) sgn ( DF(ua) ( ZSUa )- a >> sv J z ) ch s M(zsX + w° x (-p)t). 

Using Lemma 4.6.7 and Lemma 4.6.9 (ii) and (hi) of |Yee 08j with 5 = w x (—p) gives values 
for the signs, and we find that 

ch s 6 a M{zX + w° x {-p)t) = (-lyte+K+'WMch, M{zsX + w° x {-p)t) 

-{-iy^ +x ^ch s M{zX + w° x {-p)t). 
Equation (|2j) follows immediately from Theorem 13.61 

To prove ([3]), we note that by Lemma 4.6.4 and the formula before Proposition 4.6.6 of 
[Yee08j (corrected to use appropriate notation), 

ch s 9 a L(zX + w x (—p)t) = sgn.{c'l s c' z )ch s L(zsX) 

+ sgn(c"(^(-p), za v )c' z ) Y] a X ^J ch s L(yX) 

y>ys 

where < = {-l)<^-^ D F{Va) {z p^)a" zv K (-p)t, za v ) and ^ = {-l)^- zs ^D F{Va) {zsv a )a'' zaVa . 
Using Lemmas 4.6.7 and 4.6.9 of |Yee08] as before, we obtain the desired result. □ 
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Of course the formulas in this theorem are very simple if go is a compact real form. We 
will study signature character inversion for compact real forms before treating the general 
case. 



7. Signature Character Inversion for Compact Real Forms 

The goal of this section is to perform signature character inversion by induction to give 
a simple expression for ch s L(x\) as a linear combination of signature characters of Verma 
modules, significantly improving Theorem 3.2.3 of |Yee08] . 

First, we list the recursive formulas for computing signed Kazhdan-Lusztig polynomials for 
compact real forms. Substituting into Proposition 15.71 or modifying the recursive formulas 

for classical Kazdhan-Lusztig polynomials knowing that P X}y (—q) = Px,'y x (q) (see [Yee] . 
Theorem 4.6), we have: 

Lemma 7.1. Suppose Qq is compact. Then the signed Kazhdan-Lusztig polynomials may be 

computed using Px'x x {q) = 1, Px^y x (q) = if x j£ y, and the following recursive formulas 
where s = s a where a EH\: 

a) If ys > y and xs > x > y then: 



P A ' W ° ( a ) = p x ' w ° (n) 



a') If sy > y and sx > x > y then: 



P^ x ( q ) = P ^ x o (<?)• 

w^x,w^y y ^' w^sx,w^y y ^' 



b) If y > ys and x < xs then: 



p o x o (q) + qP o (q) = q > a' x q * p x (q) 

w^xs,w^y v ^ ' ^ w^x,w\y^^' ^ / j w^z,w^y^ w^x,w^z y ^' 

z£W\\z<zs 



_p x >< ( a ) 

w^XjW^ys v" ' ' 



We could also use the recursive formulas: 
a) If ys < y and x,xs < y then: 



P X xf\q)=P^ y \q). 
a') If sy < y and sx,x < y then: 

Pxf\q) = P^hq)- 

b) If y < ys and x > xs then: 

-Pxs w y " (q) + q p x!y° x {q) = q Yl a *.'»V (w) * w ~Px,'f x (g) - p£$ (q) ■ 

zGW x \z>zs 

Theorem 16.91 applied to compact real forms gives: 
Lemma 7.2. Let Qq be compact. Let A G f}* be antidominant regular, a e Ha, and let s = s a . 
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(1) If z G W\ and z < zs, then: 

ch s 6 a M(zX + w° x (-p)t) = -ch s 6 a M(zsX + w° x (-p)t) 

= ch s M(zs\ + w° x (-p)t) - ch s M(z\ + w° x (-p)t) 

(2) If z < zs, then 

ch s 9 a L(zX + w° x (-p)t) = ch s L(zsX) - V" a*'<T A „ ch s L(yX). 

' * A A"' 

Our signature character inversion formula is the following: 

Theorem 7.3. Let g be compact. Let X G f)* be antidominant and let x G W\. Then 
inverting the formula: 

ch s M(xX + w° x (-p)t) = £ P^o y (l)c/^(yA) 

y<x 

gives: 

ch s L(xX) = ^(-1)^)-%) P^°(l) cfc s M(yA + w°(-p)t). 

Proof. We prove this result by induction on x. 

If x = 1, then M(A) = L(X) while P x j x (q) = 1, and the theorem holds. 
Suppose that x > 1. By induction, assume that for all z < x, 

chsL(zX) = J2(-l) e{z) - e{y) Pjf ° (1) ch s M(yX + w° x (-p)t). 

We wish to prove the theorem for x. There exists some s = s a where a G H\, so that 
£(x) = £(xs) + 1. By the induction hypothesis, 

ch s L(xsX + w° x (-p)t) = J2(-lY {xs) - £{y) Py^ l (l) ch s M(yX + w° x (-p)t). 

y<xs 

We apply 8 a to both sides and obtain: 

ch s e a L{xsX + w° x {-p)t) = ^2(-l) t(xa) - i(v) Py X xh^ ch s 6 a M{yX + w° x {-p)t). 

y<xs 

Applying Lemma [7.21 gives: 

ch s L(xX)-Y / ^ol <y>1 ch s L(yX) = £ { -lf^ P^/(l) x 

y>ys y<xs 

e y (ch s M(yX + w° x (-p)t) - ch s M(ysX + w° x (-p)t) 

where e y = 1 if y > ys and — 1 if y < ys. Applying the induction hypothesis to ch s L(yX) 
and rearranging, we have: 

chsL(xX) = {-lY (xs) ~ m P^h^y (ch s M{yX + w° x (-p)t) - ch s M{ysX + w° x (-p)tfj 

y<xs 

+ £ ^(-i)""-*»p« A r s (i)^A/(,A +tu »(-p) t) 

y>ys z<y 
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We wish to show that the right side is equal to J2 y < x (~ 1)^ e ^ Py, , x x (l)ch s M(yX + w°(— p)t). 



Suppose z < x. We compute the coefficient of ch s M(z\ + w\{—p)t) in the right side. 
Case 1): If z < zs < xs\ the coefficient is 

-,A,lO?/., N / „ s f('rs\—t(i'n\ t->A,1U\ i- \ . \ A,1I)5 / _ \ Hli\ — (<v\ nA,lU (l 



y>ys,y>z 



(-i)^-^(p^(i)-p^i(i)+ E ^li^(i) 

V y>ys,y>z / 

\ y>ys,y>zs , 



-,\,w 



(_l)«*)-<Wp^ (i) by formula b) 
(_1)«»M«/#*(1). 



The first equality holds since a ' A x 7^ implies that £(a?) and ^(y) have the same parity, 

A «i° 

and since P Ui '^ x (q) = P u ,v(—q) for compact real forms implies that 

a ,»?^.,»9„.1 - 1 a w°xs,w° 



w°xs,w°y,l \ u w^X8,wly,l 

1) 2 ay a;s l by [Hum90j, p. 165 



X,w° 
^y,xs,l ' 



The second equality holds by formula a) and since for z < zs and y > ys, by Property Z, 
y > z if and only if y > zs. 

Case 2): If zs < z < xs: the coefficient of ch s M(z\ + w x ) (—p)t) is 

(_l)i(«MMj£?(l) + (-l)««M<">j£g(l) + £ a^^C-l^-'W/^Cl) 



V y>ys,y>z / 



y>ys,y>z 



= (-l^-^P^l) by formula b). 

Case 3): If 2 ^ xs but zs < xs: then zs < 2; and by Property Z, z < x. The coefficient 
of c/i s M(zA + u^(-p)t) is: 



y>ys,y>z 



(_1)«*WM fp*$(l) + ^ 

V y>ys,y>z 

(-i)^)-^)p^(i). 
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The second equality holds since a*'o^ s w o y 1 7^ implies that y < xs while P Z)V ^ implies 
that z < y. There are no y so that z < y < xs, so the sum is zero. 

The third equality holds by, for example, formula 4.1c) of |dC02] and by Theorem 4.6 of 
[Yee] . 

Case 4): If z < xs but zs ^ xs: then zs > z and by Property Z, zs < x. The coefficient 
of ch s M(zX + wl(-p)t) is: 

-(-i)^-^pi^ (i) + £ (-1)^-^41,^,^(1) 

= (-1)^-^^/(1) + E since P*'$ = 

= (-l)^Mj£$(l) = (-i)^)-^p^(i). 

In all cases, we see that the theorem holds for x. By induction, the theorem holds in 
general. □ 

8. Signature Character Inversion for Equal Rank Real Forms 

An easy change of basis argument extends the signature character inversion theorem from 
compact real forms to all equal rank real forms. Recall that signed Kazhdan-Lusztig poly- 
nomials for equal rank real forms and classical Kazhdan-Lusztig polynomials are closely 
related: 

Proposition 8.1. ( [Yee] . Theorem 4-6) Suppose go is an equal rank real form. Let X be 
antidominant and let x, y G W\. Signed Kazhdan-Lusztig polynomials and classical Kazhdan- 
Lusztig polynomials have the following relationship: 

p x x ; y w Hq) = (-iy (xX - vX) p x>y (-q) 

where e is the ^-grading on the (imaginary) root lattice. That is, e(/i) is the parity of the 
number of non-compact roots in an expression for p as a sum of roots. 

Therefore: 

Theorem 8.2. Let Qo be an equal rank real form, X G h* be antidominant, and let x G W\. 
Then inverting the formula: 

ch a M(xX + w° x (-p)t) = £ P^} w o y (l)ch s L(y*) 

y<x 

gives: 

ch s L(xX) = J2(-iy {x) - i{v) Pyf\l) ch s M(yX + w° x (-p)t). 

y<x 

A w® 

Proof. In the case where q is compact, P x , y A (g) = P x ,y(~ Q)- We have shown in this case 
that 

ch s M(xX + w° x (-p)t) = ^ P w o x w o y (-l)ch s L(yX) 

y<x 
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inverts to give: 

ch s L{x\) = ^(-1)^)-%) P y>x (-1) ch s M(yX + w° x (-p)t). 

y<x 

Note that in the equal rank case, 

ch s M(x\ + w° x (-p)t) = ^(-l)«^*P <x , <y (-l)ch,L(vX), 

y<x 

so the equal rank case differs from the compact case by conjugating the change of basis 
matrix by the diagonal matrix with entries (— l) 6 ^) in the (x, x)-positions on the diagonal. 
(Extend e naturally to QA r .) It thus follows that the inverse change of basis matrix also 
differs by conjugation by the diagonal matrix with (— l) e ( xX ) on the diagonal, whence 

ch s L{x\) = (-i)«**-vX)p ViX (-i) chs M(yX + w°(-p)t) 

y<x 

= Py X ;<(l) ch s M{y\ + w x (-p)t), 

y<x 



proving our theorem for equal rank real forms. □ 

9. Conclusion 

Although the classification of unitary highest weight modules has been solved by work 
of Enright-Howe-Wallach, it would be interesting to recover the classification using signed 
Kazhdan-Lusztig polynomials and the formulas in this paper and in |Yeej . Cohomological 
induction applied to highest weight modules produces Harish- Chandra modules for which 
signatures can be recorded using signed Kazhdan-Lusztig- Vogan polynomials. Techniques 
used to identify unitary representations among highest weight modules may very well have 
analogues for Harish- Chandra modules. 
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